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The  guidance  for  achieving  better  buying  power  set  forth  in  Dr.  Carter's  Sept.  14,  2010, 
memo  requires  action  by  every  member  of  the  acquisition  community.  This  article 
considers  just  two  of  the  many  processes  that  are  critical  for  program  success:  con¬ 
ducting  a  sound  program  life  cycle  cost  estimate  and  establishing  a  program's  budget. 
These  two  processes  are  interlinked  in  that  the  best  available  cost  estimate  should  be 
used  to  determine  the  program  budget,  not  only  at  major  acquisition  milestones,  but  also  an¬ 
nually  during  the  budgeting  cycle.  Interlinking  of  these  processes  is  meant  to  ensure  that  the 
resources  devoted  to  large  programs  are  sufficient  to  complete  them  successfully,  based  on  a 
comprehensive  oversight  process  that  includes  scrutiny  at  least  at  major  milestones  as  well  as 
annual  re-evaluations  as  part  of  DoD's  budgeting  process. 


Husband  is  a  professor  of  cost  analysis  at  DAU  with  16  years  of  acquisition  experience  in  cost  estimating,  systems  engineering,  and  R&D 
project  management. 
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Yet  despite  DoD's  extensive  oversight  process  and  the  best  ef¬ 
forts  of  its  workforce,  the  Department's  acquisition  programs 
are  increasingly  plagued  by  worsening  cost  and  schedule 
growth  and  failure  to  deliver  promised  performance.  While 
conducting  sound  estimates  and  appropriately  establishing 
program  budgets  cannot  ensure  successful  outcomes,  they 
are  two  key  processes  that  must  be  done  right  if  a  program  is 
to  have  any  chance  of  success. 

Sound  Cost  Estimates 

Conducting  a  sound  cost  estimate  is  a  very  difficult  task.  "It's 
tough  to  make  predictions,  especially  about  the  future,"  has 
been  variously  attributed  to  Niels  Bohr,  Mark  Twain  and,  of 
course,  Yogi  Berra.  In  his  best-selling  book  The  Black  Swan, 
Nassim  Taleb  addresses  humanity's  horrible  record  of  pre¬ 
dicting  the  future,  and  particularly  bemoans  the  fact  that,  be¬ 
cause  of  our  facility  in  inventing  stories  that  convince  us  we 
understand  the  past,  we  unaccountably  continue  to  believe 
we  can  predict  the  future  well.  He  attributes  our  poor  record 
in  predicting  to  an  inability  to  contemplate  out-of-the  ordinary 
events,  what  he  terms  "Black  Swans,"  referred  to  as  "unknown 
unknowns"  in  military  parlance. 

So  what's  my  point  in  quoting  malapropisms  about  the  future 
and  describing  unk-unks?  Because  it  is  important  to  recognize 
that  a  cost  estimate  is  a  prediction  of  the  future,  a  bet  on  what 
we  believe  is  going  to  happen.  An  estimate  needs  to  be  much 
more  than  just  a  set  of  numbers  that  are  used  to  establish  a 
program's  budget.  A  good  estimate  provides  decision  mak¬ 
ers  with  key  insights  into  the  risks  (and  opportunities)  of  a 
program.  A  broad  explication  of  technical  and  schedule  risks 
are  sine  qua  non  features  of  a  sound  estimate,  but  a  really 
good  estimate  also  specifically  identifies  key  program  risks. 
Examples  include  the  validity  of  critical  programmatic  assump¬ 
tions,  appropriateness  of  the  acquisition  strategy,  fluctuations 
in  contractor  business  base,  problems  with  outsourcing  strat¬ 
egy  or  diminishing  manufacturing/material  sources,  and  other 
Black  Swans  that  I  haven't  even  thought  of  but  that  turn  out 
(so  obviously  in  hindsight!)  to  be  important.  It  goes  without 
saying  that  identifyingall  the  key  risks  in  advance  is  really  hard. 

Eminent  statistician  George  Ball's  quote  about  models  aptly 
describes  cost  estimates:  "All  cost  estimates  are  wrong;  some 
are  useful."  This  means  that  even  though  it's  unrealistic  to 
expect  a  program  to  cost  exactly  what  is  predicted,  a  good 
estimate  has  value  in  the  information  that  it  provides  to  deci¬ 
sion  makers.  Besides  identifying  and  providing  insights  into 
risks  and  opportunities,  a  sound  estimate  must  also  explicitly 
identify  key  cost  drivers  and  quantify  them  to  an  appropriate 
degree  of  precision. 

What  is  appropriate?  In  most  cases,  the  distributions  of  an 
estimate  (or  distributions  of  the  key  cost  drivers)  presented 
to  decision  makers  should  be  broader  than  those  that  have 
been  proffered  in  the  past.  That  is,  historical  data  indicate  that 
distributions  of  outcomes  are  broad  compared  to  the  distribu¬ 
tions  predicted  in  typical  cost  estimates. 


So,  while  the  bottom-line  numbers  given  by  a  cost  estimate 
are  necessary  to  adequately  resource  a  program,  in  my  expe¬ 
rience,  the  most  important  information  that  cost  estimates 
provide  senior  decision  makers  are  insights  into  the  program's 
risks  and  cost  drivers. 

There  are  other  crucial  characteristics  of  a  sound  estimate  that 
may  not  be  explicitly  considered  by  senior  decision  makers 
but  need  to  be  considered  by  the  program  manager  and  oth¬ 
ers  charged  with  reviewing  and  assessing  the  estimate.  These 
include  the  use  of  sound  estimating  practices  and  techniques, 
such  as: 

■  Using  a  variety  of  techniques  to  crosscheck  results. 

Ideally,  data  are  derived  from  historical  actuals  that  in  all 
cases  have  been  appropriately  normalized  and  adapted  so 
that  the  data  are  applicable  to  the  program  being  costed 
(much  easier  said  than  done). 

■  Incorporating  all  available,  relevant  information  into  the 
estimate  when  it  is  presented  to  decision  makers.  This 
sounds  obvious  and  straightforward,  but  again,  it  is  easier 
said  than  done  because  of  the  long  time  frame  required  to 
produce  an  estimate  and  because  estimates  are  sometimes 
structured  in  a  way  that  makes  it  difficult  to  update  them 
quickly. 

•  Ensuring  the  estimate  is  robust.  For  instance,  it  should 
provide  the  appropriate  level  of  detail  (which  varies  de¬ 
pending  on  the  estimate's  purpose);  it  should  be  created 
by  personnel  with  sufficient  expertise  and  experience  to 
exercise  judgment  about  the  critical  factors  that  influence 
the  estimate;  and  ideally,  it  should  be  scrutinized  and  evalu¬ 
ated  by  independent,  impartial  experts.  All  these  aspects 
require  that  appropriate  time  and  manpower  be  allocated 
for  the  estimate. 

PMs  and  others  charged  with  evaluating  an  estimate  should 
ensure  that  it: 

•  Provides  explication  of  risks,  both  in  a  general  and  specific 
sense. 

•  Identifies  and  quantifies  key  drivers  using  sensitivity 
analyses  to  an  appropriate  degree  depending  on  the 
purpose  of  the  estimate  and  the  time  frame  available  for 
producing  it. 

•  Is  based  on  sound  data  that  are  appropriately  relevant  to 
the  program  under  consideration. 

•  Takes  into  account  the  most  recent  information  available 
on  the  program. 

•  Provides  a  sufficiently  robust  level  of  detail  and  has  been 
independently  scrutinized. 

A  PM  has  the  ultimate  responsibility  to  review  and  assess  the 
validity  of  a  program  office's  cost  estimate  and  to  present  it  to 
superiors  in  a  balanced,  responsible  way.  This  is  no  small  task, 
given  the  DoD  and  military  culture  of  approaching  all  problems 
and  issues  with  an  optimistic,  can-do  attitude.  Moreover,  with 
more  than  enough  to  do,  it  is  a  natural  tendency  to  believe 
good  news  about  resolution  of  potential  issues,  whether  it 
comes  from  contractors  or  program  office  subordinates.  While 
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counterintuitive,  devoting  some  time 
to  continue  to  examine  issues  that 
are  supposedly  "on-track"  is 
one  strategy  to  provide  early 
warning  about  problems 
before  they  become  un¬ 
manageable. 

A  sound  cost  esti¬ 
mate  is  a  neces¬ 
sary,  but  not  suf¬ 
ficient  condition 
in  the  process  of 
providing  a  pro¬ 
gram  with  the 
resources  neces¬ 
sary  for  it  to  be 
executed  success¬ 
fully.  Ultimately, 
program  success  will 
depend  more  on  the 
process  the  cost  esti¬ 
mate  is  meant  to  inform, 
namely,  establishing  the 
program's  budget.  Some  of  the 
reasons  program  budgets  are  often 
misaligned  with  the  program's  best  cost 
estimate  are  discussed  below. 


Misalignment  Between  Program  Budgets 
and  Cost  Estimates 

The  DoD  expends  enormous  effort  and  funds  on  program 
oversight.  MDAPs  are  notorious  in  this  regard,  but  indeed 
DoD  programs  at  all  levels  are  renowned  for  having  an  ex¬ 
traordinary  amount  of  "red  tape."  Given  the  huge  effort  and 
extensive  deliberation  associated  with  DoD's  Planning,  Pro¬ 
gramming,  Budgeting  and  Execution  process,  one  would  think 
that  programs,  after  being  approved  for  initiation  and  given  a 
budget,  would  in  general  be  adequately  resourced  to  have  a 
reasonable  chance  of  success.  Would  that  it  were  so!  There 
are  numerous  reasons  why  a  program's  established  budget 
does  not  match  the  most  reasonable  expectation  of  what  that 
program  will  most  likely  cost.  Most  disappointing  are  instances 
when  decision  makers  should  know  that  the  program's  budget 
is  likely  to  be  inadequate,  yet  it  is  still  underfunded  during  the 
PPBE  process  (see,  for  example,  GAO-11-380R,  the  March  25, 
2011,  report  on  the  Presidential  Helicopter  Program  and  other 
GAO  reports  on  this  program). 

The  full-funding  requirement  is  meant  to  ensure  that  DoD  pro¬ 
grams  have  adequate  resources  budgeted  currently  and  in  the 
out-years  to  achieve  their  approved  acquisition  strategy.  Full 
funding  is  a  DODI  5000.02  regulatory  requirement  at  mile¬ 
stones  A,  B,  and  C  and  is  a  statutory  requirement  as  part  of 
Title  10  Section  2366  certification  at  Milestone  B.  One  would 
infer  that  programs  should  remain  fully-funded  throughout  the 
acquisition  process,  with  their  budgets  adjusted  to  reflect  the 


An  estimate 
needs  to  be  much  more 


than  just  a  set  of  numbers  that 
are  used  to  establish  a  program's 
budget.  A  good  estimate  provides 
decision  makers  with  key 
insights  into  the  risks  (and 
opportunities)  of  a 


latest  best  available  cost  estimate,  but 
unfortunately  that  is  often  not  the 
case.  Until  recently,  there  was 
little  penalty  for  certifying  a 
program  as  Fully-Funded 
at  a  major  acquisition 
milestone  and  then  cut¬ 
ting  its  budget  during 
a  subsequent  budget 
cycle,  sometimes 
even  the  cycle  im¬ 
mediately  following 
certification. 


To  be  sure,  some¬ 
times  the  discon¬ 
nect  is  the  result 

/of  revised  cost  es¬ 
timates  that  predict 
cost  growth  beyond 
the  budgeted  amount. 
Or  the  disconnect  could 
result  from  poor  program 
execution,  either  in  terms  of 
delivering  what  was  promised  or 
obligating  and  executing  funds  ac¬ 
cording  to  the  planned  timeline.  However, 
programs  executing  successfully  are  also  sub¬ 
jected  to  budget  cuts  during  the  budgeting  cycle  of  PPBE  that 
cause  them  to  be  underfunded  compared  to  the  best  available 
cost  estimate,  even  when  the  DAE  has  directed  budgeting  to 
that  estimate  at  the  most  recent  milestone. 

Structural  reasons  for  this  behavior  abound:  due  to  its  insa¬ 
tiable  appetite,  the  Department  has  more  programs  on-going 
than  can  be  funded  adequately  (the  "bow  wave"  problem); 
resource  limitations  lead  some  decision  makers  to  rationalize 
that  modest,  recurring  budget  cuts  drive  efficiency  by  eliminat¬ 
ing  waste  and  non-value  added  work  (when  in  fact  they  impair 
efficiency  by  subjectingthe  program  to  a  "death  of  a  thousand 
cuts");  and,  the  separate  authorities  and  prerogatives  of  the 
requirements,  acquisition,  and  PPBE  communities  mean  that 
acquisition  "decisions"  are  not  necessarily  resourced  in  the 
budgeting  cycle.  Finally,  over-optimism  can  derail  a  program  at 
any  time  from  birth  to  infancy  to  adolescence— during  the  cost 
estimation  process;  during  the  milestone  decision  if  a  lower, 
optimistic  cost  estimate  is  selected  as  the  basis  for  the  APB; 
and  during  program  execution,  when  our  optimistic  culture 
inhibits  PMs  and  decision  makers  from  recognizing  and  re¬ 
sponding  to  problems  that  arise  within  programs. 

The  Expeditionary  Fighting  Vehicle  (EFV),  recently  recom¬ 
mended  for  cancellation  by  then-Secretary  Gates,  is  an  in¬ 
structive  example  of  a  program  in  which  institutional  failures 
related  to  cost  estimating  and  resourcing  occurred  that  in 
hindsight  seem  obvious.  (As  I  also  fall  squarely  into  the  trap 
identified  by  Taleb  of  inventing  a  story  that  perfectly  explains 


47 


Defense  AT&L:  Better  Buying  Power  •  September-October  2011 


events— after  the  fact).  Initial  cost  estimates  for  the  SDD 
phase  of  the  program  in  December  2000  at  Milestone  II  by 
the  Service  and  OSD/CAIG  were  $0.86  billion  and  $1.24  billion 
in  base  year  2007  dollars,  respectively.  The  MDA  elected  to 
baseline  the  program  based  on  the  service  cost  position,  which 
was  considerably  more  optimistic  than  the  CAIG's. 

During  the  EFV's  Nunn-McCurdy  breach  certification  pro¬ 
cess  in  2007,  other  estimating  and  resourcing  shortcomings 
emerged  that  contributed  to  the  program's  cost  growth  and 
failure  to  meet  KPPs.  One  was  that  the  original  cost  esti¬ 
mates— both  by  the  Service  and  the  CAIG— were  primarily 
based  on  analogies  to  the  Bradley  Infantry  Fighting  Vehicle  and 
previous  tank  programs.  The  good  news  was  that  abundant 
cost  data  existed  on  those  historical  programs.  The  bad  news 
was  that  the  technological  complexity  of  the  EFV  made  these 
historical  programs  poor  analogies,  both  for  the  RDT&E  and 
the  Procurement  phases  of  the  EFV  program. 

This  is,  unfortunately,  a  common  problem  cost  estimators  face 
when  estimating  revolutionary  (vice  evolutionary)  programs, 
such  as  weapon  systems  that  are  invisible  to  the  enemy,  heli¬ 
copters  that  can  fly  like  aircraft,  tanks  that  can  ski  across  the 
water,  and  virtually  all  space  and  satellite  programs.  On  the 
resourcing  side,  like  many  DoD  programs,  EFV  suffered  budget 
cuts  that  forced  scaledown  from  the  originally  designed  SDD 
program.  Among  the  casualties  of  these  budget  cuts  was  a  Re¬ 
liability  Improvement  Program  proposed  by  the  contractor  to 
improve  subsystem  reliability.  Although  failure  to  meet  the  reli¬ 
ability  KPP  threshold  was  the  key  cause  of  cost  growth,  which 
drove  quantity  reductions,  which  drove  further  cost  growth 
and  ultimately  put  EFV  into  a  Nunn-McCurdy  breach  situation, 
it  would  be  disingenuous  to  assert  that  better  cost  estimating 
and  resourcing  alone  would  have  prevented  program  failure. 

The  oft-occurring  bugaboos  of  technical  immaturity  and  its 
corollary,  over-optimistic  timelines,  are  also  cited  by  the  GAO 
as  contributing  factors  to  EFV  cost  growth.  Still,  sound  es¬ 
timates  and  good  resourcing  are  meant  to  take  challenging 
timelines  and  the  state  of  technology  into  account.  So  when 
a  program  fails,  it  is  instructive  to  begin  at  the  beginning  and 
question  the  soundness  of  the  estimate  and  the  resourcing 
decisions,  while  heeding  Taleb's  admonition  that  stories  we 
invent  in  hindsight  that  neatly  explain  events  may  not  have 
been  perceivable  in  advance. 

Conclusion 

So  what  can  a  PM  do  to  improve  the  cost  estimating  and  re¬ 
source  allocation  processes  for  his  or  her  program?  At  first 
blush,  it  seems  like  an  insurmountable  task,  as  cost  estimates 
that  established  the  program  baseline  may  have  been  done 
years  earlier  and  budget  cuts  are  a  systemic  feature  of  our  sys¬ 
tem.  The  advice  I  offer  falls  squarely  into  that  common-sense, 
non-profound  set  of  good  management  principles  that  all  PMs 
are  doing  their  best  to  adhere  to  every  day.  Be  skeptical.  Be 
transparent.  Be  resolute  and  courageous. 


Be  Skeptical 

When  evaluating  your  cost  estimate,  be  skeptical  in  a  big- 
picture  sort  of  way.  In  other  words:  As  PM  you  don’t  have  the 
time  or  even  necessarily  the  expertise  to  second-guess  cost 
estimating  techniques  and  methodologies  or  the  accuracy  and 
validity  of  data  sources.  But  it  is  within  your  purview  to  create 
an  environment  in  which  key  assumptions  and  even  require¬ 
ments  are  regularly  re-evaluated  to  ensure  they  are  still  valid. 
They  may  have  been  valid  at  the  time  they  were  established. 
But  conditions  change,  and  sometimes  requirements  prove  to 
be  impossible  to  meet,  and  the  sooner  those  changed  condi¬ 
tions  are  recognized  and  dealt  with,  the  better  for  your  pro¬ 
gram.  If  you  are  lucky  enough  to  be  the  PM  at  program  initia¬ 
tion,  wargame  the  cost  estimate  as  if  you're  going  into  battle 
and  your  ASR  depends  on  it.  (Ok,  APB  is  the  better  acronym, 
but  I  couldn't  resist.) 

Be  Transparent 

In  all  things,  be  transparent.  Again,  this  is  an  obvious  positive 
attribute,  but  here's  what  I  mean  in  this  context:  When  you 
become  aware  of  issues  that  negatively  affect  your  program's 
cost,  rapidly  gather  the  information,  alternatives,  and  proposed 
solutions  related  to  those  issues  so  that  your  leadership  and 
decision  makers  in  charge  of  resources  are  informed  as  soon 
as  possible.  There  is  a  tendency  in  our  system  to  avoid  surfac¬ 
ing  a  new  cost  estimate  that  predicts  cost  growth,  particularly 
as  the  Nunn-McCurdy  breach  thresholds  are  approached.  Do 
your  best  to  ensure  that  program  issues  get  timely  attention, 
despite  the  delays  inherent  in  our  process. 

Be  Resolute  and  Courageous 

Be  resolute  and  courageous  about  the  resources  required  for 
your  program.  I  realize  this  is  far  easier  to  say  than  do,  and  the 
reality  of  our  system  is  that  PMs  are  expected  to  take  their  "fair 
share"  of  cuts  and  still  do  the  job  originally  promised.  Some 
people  say  it  demonstrates  a  lack  of  credibility  to  say  "If  you 
cut  my  program  by  5  percent,  you  might  as  well  cancel  it."  My 
view,  in  contrast,  is  that  it  is  in  the  best  interest  of  the  program 
and  the  DoD  to  quantify  to  the  best  degree  possible  the  effect 
of  cuts  and  change  the  program  accordingly,  whether  that  be 
by  reducing  requirements  or  by  extending  the  program  time¬ 
line,  with  the  associated  increased  out-year  expenditures  that 
entails.  And  if  a  5-percent  cut  in  a  program  this  year  is  going 
to  result  in  a  requirement  for  three  times  that  amount  of  funds 
in  the  future  (a  conservative  estimate!),  those  effects  should 
be  documented  and  provided  to  decision  makers.  To  be  sure, 
such  an  approach  requires  courage  because  it  is  a  departure 
from  how  the  Department  has  done  business  in  the  past,  when 
cost  growth  was  more  or  less  accepted  as  part  of  the  process. 
Our  belated  realization  that  affordability  must  be  on  an  equal 
par  with  performance  necessitates  that  we  make  changes  to 
that  past  way  of  doing  business,  so  that  we  produce  affordable 
systems  with  acceptable  rather  than  exquisite  performance 
within  reasonable  timelines  to  support  our  warfighters. 

The  author  can  be  reached  at  mark.husband@dau.mil. 
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